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Purpose: Emergency Medicine (EM) residents take the American Board of Emergency 
Medicine (ABEM) in-training exam, and perfomiance on this exam has been shown to correlate 
to performance on the ABEM qualifying exam. Though many residencies have in-training 
exam preparation activities, there is little data on the effectiveness of these efforts. This study 
aimed to elicit resident perspectives about the exam and exam preparation in order to generate 
hypotheses and better inform future preparation efforts. 

Methods: Second- and third-year EM residents at a single institution were interviewed using 
a semi-structured format. Qualitative methodology was used to analyze the data. 
Results: Thirteen EM residents participated in the study. Eight major themes and 1 8 sub-themes 
were identified. These were further characterized as relating to the exam itself or to exam 
preparation. Residents generally value the in-training exam. Sixty-nine percent noted that it 
provided an assessment of their current knowledge and deficiencies. Thirty-eight percent noted 
that it improved familiarity with the qualifying exam. Regarding exam preparation, residents 
stated that a question format was preferred, especially when accompanying explanations were 
of high quality. Additionally, practical considerations, such as portability, impacted resident 
selection of study tools. 

Conclusion: Residents value the in-training exam as a marker of their academic progress 
and for their ability to gain familiarity with the qualifying exam. They prefer question-based 
preparation over text-based learning, as long as there is a detailed explanation of each answer. 
Educators creating structured in-training review may want to focus on question-based material 
with detailed explanations. 

Keywords: examination preparation, graduate medical education, in-training examination, 
resident education 

Introduction 

Emergency medicine (EM) residents take the American Board of Emergency Medicine 
(ABEM) in-training exam annually. Performance on this test has been shown to 
correlate with performance in the ABEM qualifying exam, which is required for board 
certification.' Optimizing performance on the in- training exam may therefore be of 
value to resident physicians and program directors alike. 

The best method of preparation for in-training exams is currently unclear, and 
programs to aid in preparation have reported mixed results.^"* Cheng found that an EM 
residency-developed review course did not affect performance on the in-training exam.^ 
Gillen also failed to demonstrate improved exam performance after implementation 
of a program that utilized structured reading and graded quizzes.' However, there is 
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some evidence from other specialties that preparation can in 
fact improve scores on in-training exams.''^* 

To better direct future attempts at exam preparation, it is 
worth considering the views of the trainees. Residents' opin- 
ions regarding the exam itself and strategies for preparation 
are unknown. The aim of this investigation is to characterize 
resident perspectives on the exam, with the hope that infor- 
mation can be obtained that may aid in the development of 
more effective review methods. 

Material and methods 

Research design 

Semi-structured interviews of senior EM residents at one 
institution were performed. Data analysis used qualitative 
methodology in order to allow generation of hypotheses. 
The study was reviewed by The Office of Responsible 
Research Practices (ORRP) at The Ohio State University 
(OSU) and it was deemed exempt from Institutional Review 
Board review. 

Study setting and population 

Study participants were second- and third-year EM resident 
physicians at OSU. Residents at this program are required 
to take the exam annually. The study took place between 
March 2012 and May 2012. These groups of residents, and 
time of year, were selected to ensure that all participants had 
taken the exam at least two times. Residency education at 
OSU follows the requirements of the Accreditation Council 
for Graduate Medical Education. 

Twenty-four residents ( 1 2 per year) were asked to partici- 
pate by an email request for an interview. Participation was 
voluntary, and this was made clear in the email, as required 
by ORRP. Prior to beginning each interview, written con- 
sent to participate was obtained, and further assurances of 
confidentiality were provided. 

Study protocol and data analysis 

An interview guide was created by the investigators in 
advance of this study. The first version was reviewed and 
revised, based on feedback from other staff of the EM 
department with experience in residency education. The 
final version consisted of five questions that were intended to 
stimulate conversation (Table 1). Interviewers were encour- 
aged to ask additional follow-up questions to maximize 
response and to promote discussion. One-on-one interviews 
with residents were conducted. These interviews were 
performed by one of two investigators, based on schedule 



Table I Guide used in interviews 

1 . How do you feel about the in-training exam! 

2. Discuss the importance of the in-training exam and preparing for it. 

3. What methods have you used in the past to prepare for the exam? 
Why? 

4. What is important to you when choosing a method for exam 
preparation? 

5. What would be your ideal method of preparation for the in-training 
exam? 



compatibility with the resident. Interviewers recorded notes 
during and immediately after the conversation. On a separate 
page, descriptive data were also recorded, including resident's 
year in training, and sex. 

Interview notes, and separate descriptive data, were stored 
securely until all interviews were completed, at which point 
they were analyzed using basic descriptive statistics and 
qualitative methodology.'^ The investigators independently 
reviewed the interview notes (both the notes they took and 
the notes the other investigator took). As themes and sub- 
themes emerged during reading, they were recorded. After 
completion of thematic analyses, the two investigators met 
to compare results. Differences were discussed and a final 
consensus thematic analysis was agreed upon. The investi- 
gators then went back to review all the notes to determine 
the frequency of themes and sub-themes collaboratively. 
If the investigators agreed that a single response directly 
related to more than one theme, it was coded as such. In 
some instances, responses clearly fit within a major theme, 
but not a sub-theme, and were simply counted as relating to 
the major theme. 

Results 

Thirteen residents agreed to participate in this study. Six 
of the participants were third- year residents (two female, 
four male) and seven were second-year residents (four 
female, three male). The mean USMLE Step 1 score for 
the thirteen participants was 220. The mean USMLE Step 
2 CK score was 235. One investigator (TRE) conducted 
six interviews and seven were conducted by the other 
(AWB). 

A total of eight major themes were noted. Within these 
major themes, eighteen sub-themes were noted. Major themes 
and sub-themes, as well as the frequencies of each, are shown 
in Tables 2 and 3. The major themes could be further char- 
acterized within domains of attitudes regarding the exam, 
and opinions regarding preparation. We further describe the 
themes and sub-themes below. 
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Table 2 Thematic content of attitudes and opinions regarding 



the in-training exam 


Major theme 


Sub-theme 


Frequency 


Value 




2! 


examination 


Motivation to study 


3 




Provide assessment of current 


9 




l<nov/ledge/deficiencies 






Monitor progression over training years 


3 




Compare results to classmates/peers 


2 




Improved familiarity w\th ABEM exam 


5 


Weaknesses 




2 


of exam 


Score reporting not immediate 


1 




No ability to review questions 


1 



Abbreviation: ABEM, American Board of Emergency Medicine. 



Resident attitudes about 
the in-training exam 

The two major themes in this category were "value examina- 
tion" and "weaknesses of exam". There were 21 responses 
in the value examination theme and two responses in 
the weaknesses of exam theme. Within the theme value 

Table 3 Thematic content of opinions regarding in-training exam 



preparation 

Major theme Sub-theme Frequency 

Question format 20 
optimal Similar to ABEM format 4 

Patient centered/clinically 2 

relevant 

Answers provide enough detail 5 

Provides assessment of I 

strengths/weaknesses 

Text-based learning less 4 

valuable 

Practical aspects I I 

important Portable 3 

User-friendly 3 

Affordable I 

Ability to set pace, 4 

small increments, affect 

method selection 
Preparation timing 8 

Closer to ABEM exam 4 

more preparation 

Study weeks/months 4 
before exam 

Preparation should 7 
be relevant To exam performance S 

To clinical practice 2 
Peer/fellow resident No sub-themes 6 
recommendations 
highly regarded 

Longitudinal study ideal No sub-themes 2 
Abbreviation: ABEM, American Board of Emergency Medicine. 



examination, five sub-themes emerged, the most common 
of which was "provides assessment of current knowledge 
and deficiencies". 

Value examination 

The most commonly coded theme was that residents value 
the in-training examination. Within this major theme, five 
sub-themes were identified. Nine of the thirteen participants 
(69%) noted that the exam "provides an assessment of their 
current knowledge and deficiencies". Thirty-eight percent 
mentioned an "improved familiarity with the ABEM quali- 
fying exam" (the "real" exam) as a reason for valuing the 
in-training exam. Other sub-themes included "motivation to 
study", "monitoring progression over the course of training", 
and "the ability to compare him/herself to peers". 

Weaknesses of examination 

Mention of in-training exam weaknesses was noted in two 
interviews. One participant noted that "score reporting was 
not immediate" and another noted the "lack of ability to 
review the questions" after the exam. 

Resident opinions regarding in-training 
exam preparation 

In this category, six major themes emerged: "question format 
is optimal", "importance of practical aspects", "prepara- 
tion timing", "relevance of preparation", "fellow resident 
recommendations are highly regarded", and "longitudinal 
study is ideal." Of the major themes, "question format is 
optimal" (five sub-themes) was most common, with twenty 
such responses. The next-most common theme was "impor- 
tance of "practical aspects" (four sub-themes), followed by 
"preparation timing" (two sub-themes). 

Question format optimal 

Regarding resident opinions about in-training exam prepara- 
tion, the most common major theme was that residents found 
studying materials in question form to be ideal. Five partici- 
pants noted that this was true especially when the study mate- 
rials included explanations with detail sufficient to enhance 
learning. Four participants preferred a question format, 
similar to the format to the in-training and qualifying exams. 
Four residents specifically noted that text-based learning was 
less valuable. Other sub-themes included that question-based 
studying was patient-centered or clinically relevant (two resi- 
dents), and that this format provided a method of assessing 
strengths and weaknesses (one resident). 
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Practical aspects important 

The second most common theme regarding exam prepara- 
tion was that there are practical issues that residents consider 
when choosing a study method. Among these practical con- 
cerns, the ability to set their own pace and to study in small 
increments was noted by four participants. The portability 
and user-friendliness of materials were each noted by three 
residents as being important factors in choosing a method of 
preparation. One resident mentioned that the affordability of 
the study tool was a consideration. 

Preparation timing 

The timing of in-training exam preparation emerged as a 
major theme and was noted eight times. Within this major 
theme, two sub-themes were mentioned. Four residents noted 
that they devoted more time to in-training exam preparation 
when they were closer to taking the ABEM qualifying exam 
(ie, they studied more as senior residents than they did as 
interns). Four residents also identified that they focused on 
in-training exam preparation in the weeks and months prior 
to the exam, rather than throughout the year. 

Preparation should be relevant 

Five residents noted the importance of study practices that are 
relevant to performance in the in-training exam. Additionally, 
two residents mentioned the need to have study methods that 
are relevant to clinical practice. 

Fellow resident recommendations 
are highly regarded 

Six residents stated that they used the recommendations of 
other residents when deciding how they would prepare for 
the in-training exam. 

Discussion 

The aim of this investigation was to characterize EM resident 
attitudes regarding the in-training exam and opinions regard- 
ing optimal preparation strategies. The hope was to uncover 
information that could be used to develop more effective test 
preparation programs. The two most notable findings of this 
investigation are that residents generally perceive value in 
the annual examination and that they prefer question-based 
preparation. 

Not surprisingly, residents cited improved familiarity 
with the ABEM qualifying exam as a benefit of the in- 
training exam. However, this was not the most common 
sub-theme noted. The majority of residents mentioned using 
the results of the in-training exam as an assessment of their 



current knowledge and deficiencies. (It should be noted that 
exam results are accompanied by a detailed breakdown of 
performance in multiple topics.) Additionally, participants 
noted that the in-training exam allows them to monitor their 
progress as they advance through residency. This indicates 
that residents, undoubtedly motivated to pass the qualifying 
exam, place high value on this objective measure of their 
professional growth. Presumably, this is related to a desire 
to improve, which is also highlighted in other major themes 
and sub-themes. One resident noted that a question-based 
study method was preferred because it provided an assess- 
ment of strengths and weaknesses. Additionally, one person 
mentioned the inability to review questions after the exam 
as a weakness of the in-training exam. 

Regarding preparation for the exam, residents overwhelm- 
ingly indicated that they preferred a question-based format. 
This may be somewhat intuitive, as it mirrors the qualifying 
exam itself, and four residents mentioned this as a reason for 
choosing this manner of study. However, it seems that this 
is only true if the study material provides explanations that 
are detailed enough to enable learning, indicating that not all 
question-based materials are necessarily valued. It is note- 
worthy that almost half of the residents in this study used the 
recommendations of other residents in selecting their method 
of preparation. While it is possible that this contaminated our 
results somewhat, we believe that it is more likely that the 
promotion of question-based study aids has trickled down 
from graduating residents over several years. This may relate 
to the work of Bull et al, who found that resident-led review 
sessions were superior to faculty-led sessions." 

Interestingly, four residents specifically mentioned that 
text-based study was less valuable as it pertains to exam 
preparation. Previous work by Gillen showed that a struc- 
tured reading program had no effect on exam scores across 
all classes in an EM residency, though it did demonstrate 
improvement in postgraduate year 1 (PGY-1) scores, com- 
pared to historical controls.^ However, this study included 
a question-based component with faculty-led explanations. 
This finding may be consistent also with prior work that 
showed no improvement in in-training exam scores fol- 
lowing a didactic review course.^ Accordingly, attempts by 
residency administration to develop structured in-training 
review perhaps should be question-focused. This would 
allow not only for the review of relevant material, but also 
for discussion of test taking strategies that may improve 
exam performance. 

The timing of exam preparation was identified as being 
an important factor in exam preparation, and was specifically 
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mentioned by several residents. Not only does it appear that 
residents study more as they near the date of the exam, but 
also they seem to view the in-training exam more seriously 
as they near the qualifying exam. This may indicate that, as 
residents progress through the training program, they become 
more aware of the importance of the qualifying exam, and, as 
noted above, of the need to use their in-training exam results as 
a predictor of their future performance on the qualifying exam. 
Lastly, though four residents noted preparing in the weeks 
leading up to the in-training exam, two cited longitudinal study 
as being ideal. Due to the design of the study, it is unclear if 
this is simply a difference in approach between residents, or 
the difference between "ideal" and "actual" preparation. 

Bom from 1981 onwards, themillenial generation should 
now make up a large portion of medical trainees. Though 
we did not collect this data, it is likely that a majority of our 
sample was composed of members of this generation. Mille- 
nials are widely accepted as multi-taskers who are comfortable 
with technology, so it is no surprise that portability and user- 
friendliness of materials, and the ability to set their own pace 
were all mentioned with similar frequency by participants. 
More than one resident preferred to use electronic, question- 
based materials for exam preparation. For example, recently 
presented work utilizes the Twitter social media format as a 
method for disseminating board-review questions to residents, 
although no outcome data is available.' The preference for 
electronic format will likely become more prevalent as the 
millennial generation enters residency training en masse, and 
as more electronic study aids become available. 

While we identified several themes related to resident atti- 
tudes and opinions regarding exam preparation, we do not have 
data on the effectiveness of the preferred study methods. Some 
prior work in other specialties has shown that directed study can 
influence in-training exam performance and board pass rates.*^ 
To our knowledge, no previously published studies in EM have 
shown any specific exam preparation technique to be superior. 

Limitations 

As this was a single-center study, it is possible that resident 
attitudes and opinions simply reflect the local culture, and 
are not generalizable. This is especially relevant, since resi- 
dents cited recommendations from other residents as a key 
component to selecting a method of study. Additionally, some 
residency curricula emphasize in-training exam preparation 
more than others; the degree to which our program empha- 
sizes this may limit the generalizability of our findings. 

We utilized a semi-structured interview with pre- 
determined questions intended to stimulate conversation 



with subjects. To ensure completeness, EM faculty staff vetted 
these questions, and efforts were made during interviews to 
allow subjects to freely discuss any aspect of their exam prepa- 
ration they wished to. However, it is possible that the questions 
themselves may have biased subjects by making them believe 
that only topics raised by interviewers were important. Our 
methodology called for the authors to code interviews and to 
discuss differences of opinion. The lack of other reviewers 
to serve as "tiebreakers" may have introduced bias, although 
there were only minor discrepancies in coding. 

Finally, just over half of the eligible residents participated 
in the study. It is possible that residents with stronger opinions 
of the in-training exam were more likely to agree to par- 
ticipate. If the remaining eligible residents had participated, 
it is conceivable that this would have affected the results. 
However, the authors believe that theoretical saturation was 
achieved in the work.' ** 

Conclusion 

To our knowledge, this is the first study to identify themes 
concerning residents' attitudes and opinions about in-training 
exam preparation. Residents generally value the exam as a 
marker of their academic progress, and as an opportunity 
to gain familiarity with the qualifying exam. They prefer 
question-based preparation to text-based learning, as long as 
there is a detailed explanation of each question. Residency 
leaders who develop structured in-training review pro- 
grams can focus on question-based material with detailed 
explanations. Further work is required to overcome the 
limitations of this study, such as a multi-centered qualitative 
study, or outcomes-based quantitative evaluation. 
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